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METHOD FOR RECOGNIZI NG OBJECTS IN DIGITIZED IMAGE S 
Field of the Invention 

The present invention relates to a method for automatically recognizing one or a plurality of 
structures and/or one or a plurality of objects in digitized image data. 



Methods for automatically recognizing an object in digitized image data are known in the 
prior art. 

DE 44 06 020, for example, discloses a face recognition method. According to this method 
so-called jets are extracted from a digitized image with Gabor filters of different magnitudes 
and orientations, said jets being arranged at the nodes of a grid which is adapted to be 
subjected to displacement, scaling and deformation. This graph , i.e. the structure of the 
grid and the jets associated with the nodes of the grid, are compared with a reference graph 
comprising the structure to be recognized. For this purpose, the optimum form of the grid is 
determined by a two-phase optimization of a graph comparison function. In the first phase, 
the size and the position of the graph are optimized simultaneously; in the second phase, 
the intrinsic form of the graph is optimized. 

It has, however, turned out that a successful recognition of a structure or of an object by 
means of this method depends to a very high extent on the quality - especially on the na- 
ture of the background - of the image data comprising the structure to be recognized. It is 
true that this method is suitable for achieving good results, when the structure and the ob- 
ject have been recorded in front of a neutral background, but in cases of use where it is not 
possible to record the image data in front of a neutral background, problems may arise in 
the case of the known method; these problems may, in the final analysis, result in an un- 
satisfactory recognition of structures and objects. 



Prior art 



In the field of face recognition a further method is known in the case of which the compari- 
son between the image of a head recorded with a video camera and a plurality of images of 



heads stored in a data base is realized by a flexible mapping mechanism, the optimum 
mapping being realized by an optimization method (cf. Lades et a!., IEEE Transactions on 
Computers, 42, 300-31 1 ). 

This method is, however, disadvantageous insofar as it does not seem to be suitable for 
processing large quantities of data. It is true that Lades et al. was able to recognize an im- 
age of a head from a data base which comprised images of 87 persons, but in many cases 
of use much larger reference data bases have to be reckoned with. 

In addition, the method according to Lades et al. was only realized with a special hardware 
configuration, viz. with transputers, Le. with a plurality of microprocessors which are inter- 
connected in a predetermined manner. 

£' 
i 

j In view of these disadvantages of the methods according to the prior art, it is the object of 
I the present invention to improve the known methods in such a way that their robustness 

j with respect to less optimum image data is increased in comparison with the known 
1 method, and to guarantee in addition that the method can be realized by conventional 
■j means, 

i Description of the invention 

The above-mentioned object is achieved by a method for automatically recognizing one 
or more structures in digitized image data, said method comprising the steps of. 

(a) providing at least one reference graph comprising digitized reference image data of 
corresponding reference images, the reference graph or each reference graph compris- 
ing a net-like structure, the respective net-like structure being defined in that specific 
reference image data have assigned thereto nodes which are interconnected by links in 
a predetermined manner, and jets, each node having a jet assigned thereto and each jet 
comprising at least one sub-jet which is determined by convolutions of at least one class 
of filter functions with different magnitudes and/or orientations with the reference image 
data of the corresponding reference image at the specific node, or by convolutions of at 
least one class of filter functions with different magnitudes and/or orientations with col- 
our-segmented reference image data of the corresponding reference image at the spe- 
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cific node, or by colour information on the reference image data at the specific node, or 
by texture descriptions of the corresponding reference image at the specific node, said 
texture descriptions being gained by statistical methods, or by motion vectors at the spe- 
cific node, said motion vectors being extracted from successive reference images, 

(b) determining an optimum image graph from the digitized image data for each refer- 
ence graph/said optimum image graph representing for a specific reference graph the 
optimum adaptation to said reference, graph and being determined by projecting the net- 
like structure of said specific reference graph into the image data whereby the structure 
of the image graph is defined, and determining sublets of the image graph at the nodes 
defined by its structure, said sub-jets corresponding to at least part of the determined 
^ sub-Jets of the specific reference graph, and the projection of the net-like structure of said 
va specific reference graph being varied until a graph comparison function which compares 
,.J5 the jets of the image graph with the corresponding jets of said specific reference graph 

"U becomes optimal, 

CQ 

u * (c) associating the structure or each structure with the reference image corresponding to 
□ the reference graph for which the graph comparison function is optimal with respect to 
:;~ the optimal image graph determined for said reference graph. 

By means of this method It is possible to utilize image information of different qualities 
simultaneously, such image information being e.g. convolutions of the class/classes of 
filter functions with arbitrary magnitudes and/or orientations with the image data and/or 
convolutions of the class/classes of filter functions with arbitrary magnitudes and/or ori- 
entations with colour-segmented image data and/or colour information on the image data 
and/or image-data texture descriptions gained by statistical methods and/or motion vec- 
tors extracted from successive images. This has the effect that, in comparison with the 
prior art, an improved structure recognition rate can be realized. 

In addffion, a higher robustness in comparison with the methods known from the prior art 
can be achieved by combining image information of different qualities. 

According to a further development of the present invention a plurality of reference 
graphs can additionally be provided, and the reference graphs which have net-like 
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structures that are topologicaliy identical, i.e. which differ only with respect to the lengths 
of respective corresponding links, can be combined to form a reference bunch graph. 
Such a reference bunch graph comprises a net-like structure defined by nodes which 
correspond to the nodes of the reference graphs and by links which are determined by 
averaging the corresponding links of the reference graphs, and it further comprises 
bunch jets, each of said bunch jets being composed of the sub-jets corresponding to the 
jets at the respective nodes of the reference graphs combined in the reference bunch 
graph. In addition, an optimum image graph for the or for each reference bunch graph 
is determined according to this further development Said optimum image graph repre- 
sents for a specific reference bunch graph the optimum adaptation to said reference 
bunch graph and H is determined by projecting the net-like structure of said specific 
reference bunch graph into the image data whereby the structure of the image graph 
<*Cj is defined, and by determining sub-jets corresponding to at least part of the sub-jets 
ijl which have been used for determining the sub-jets of the reference graphs underlying 
J the specific reference bunch graph. In addition, the projection of the net-like structure of 
q said specific reference bunch graph is varied until a graph comparison function which 

rf'i 

• ' compares the jets of the image graph with the corresponding bunch jets of said specific 
Q reference bunch graph becomes optimal, sublets of the image graph being compared 

with the sub-jets in the corresponding bunch jet of said specific reference bunch graph. 
■£< Finally, each structure is associated with the reference image corresponding to the ref- 

erence graph or to the reference graph from the reference bunch graph or graphs for 

which the graph comparison function is optimal with respect to the optimal image graph 

determined for said reference graph. 

The use of reference bunch graphs permits, on the basis of the same number of refer- 
ence images, an increase in the amount of structures which are available for the com- 
parison or t in other words, a reference bunch graph permits the representation of a com- 
plex-structure object class with a small number of examples. In addition, the use of refer- 
ence bunch graphs permits such structure object classes to be modelled by examples of 
individuals. 



Depending on the respective case of use, i.e. on the structures to be recognized, only 
part of the reference graphs provided may be combined so as to form one or a plurality 
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of reference bunch graphs. Special cases within one object class can be dealt with sepa- 
rately or left out of consideration in this way. 

Alternatively, it is also possible to combine all the reference graphs provided so as to 
form one or a plurality of reference bunch graphs. 

The above-described methods can be further developed in such a way that the structure 
of the jets associated with the nodes, which is determined by the sub-jets, depends on 
the respective node. 

An a priori knowledge of the structures to be recognized can be utilized in this way. For 
^ example, specific image information can be evaluated only within a range in which it is 
actually significant Furthermore, e.g, edge filters (cut-off filters) can be used at the edge 
of the structure. 

ft? 

q Alternatively, the structure of the node-associated jets, which is determined by the sub- 
Sfl jets, can be identical for all nodes. 

Oj 

I J This further development is especially characterized in that it has a homogeneous data 
,l| structure. Hence, the method can be realized by a comparatively simple hard- and/or 
h- software implementation. 

In the above-described methods a graph comparison function can advantageously be 
used, said graph comparison function comprising a jet comparison function that takes 
into account the similarity of the jets corresponding to one another. 

In addition, the graph comparison function can comprise a comparison function for the 
net-like structure, which takes into account the metric similarity of the image graph and 
the corresponding reference graph or the corresponding reference bunch graph. In this 
case, it will be expedient to define the graph comparison function as a weighted sum of 
the jet comparison function and of the comparison function for the net-like structure. 

The jet comparison function can be defined as a function of single jet comparison func- 
tions of jets corresponding to one another. 
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For this purpose, the jet comparison function can advantageously be defined as a 
weighted sum of the single jet comparison functions and/or as a weighted product of the 
single jet comparison functions. 

in accordance with an expedient embodiment, sub-jets of the corresponding jets can be 
taken into account for determining a single jet comparison, and a single jet comparison 
function can be defined as a function of sublet comparison functions. 

In accordance with an advantageous embodiment, the single jet comparison function can 
be defined as weighted sum of the sub-jet comparison functions and/or as a weighted 
product of the sub-jet comparison functions. 

; ? | In particular , it is also possible to use different node-dependent jet comparison functions 
j y and/or single jet comparison functions and/or sub-jet comparison functions. 

Cfi In connection with the above-described reference bunch graphs, the bunch jets of the 

□ reference bunch graph B M can be divided into sub-bunch jets jjf , and the jet compari- 

■XI 

I** son function between the sub-bunch jets j" of the reference bunch graph and the cor- 
q responding sub-jets j„' of the image graph G' for n nodes for m recursions can be 
calculated according to the following formulae: 

S Jef (B M ,G) = X«>nS n {BS , ,J n , ),or 

II 

Sjet(B M ,G') = n(S„(Bjf, J,'))*" , wherein 

n 

© „ is a weighting factor for the n-th node n, and the comparison function S n (B%, J„* ) 

for the n-th node of the reference bunch graph with the n-th node of the image graph 
is given by: 



S(BV) = fi({s u (brj/)})=:n(M), with 
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« <0) (M) = XfflA <1, (M i (1) ) I or 

i 

« (0l (M) = n(n i <1> (M< 1 >)f ,or 
n (0> (M) = mp{cD i n< 1) (M< 1) )},or 

£1 <0, (M) = minfo^fNf)} , wherein (jNf = M 



^m-D^Dj = 2© J Q{-")(Mj m) ) , or 
i 

3 nT'W) = n(n< m) (M{ m >■))■■ , or 

Kj n, ( ^ t) (M i (m - 1, ) = max{a) i (ro) (Mj' n) )} 1 or 

!;| n^fMT*) = m.h{a> ^(MW)} , wherein (JMT = Mf"-* and with 

: ; 2 s(b M ,f)=2;a, n s n (Cf), 0 r 

S s(b M ,n=n(Sn(j"r)r.or 
a V 

''\ S(b M ,j-) = max{a> n S n (jrJ)},or 
S(b M ,j') = min{a) n S n (C]'}}- 

In this case, the sub-bunch jets of the reference bunch graph or graphs may comprise 
only features which have been determined by convolutions of at least one class of filter 
functions with different magnitudes and/or orientations with the reference image data of 
the corresponding reference image at the specific node, or by convolutions of at least 
one class of filter functions with different magnitudes and/or orientations with colour- 
segmented reference image data of the corresponding reference image at said specific 
node, or by colour information on the, reference image data at said specific node, or by 
texture descriptions of the corresponding reference image at said specific node, said 
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texture descriptions being gained with statistical methods, or by motion vectors at said 
specific node, said motion vectors being extracted from successive reference images. 

Alternatively, the sub-bunch jets of the reference bunch graph or graphs may comprise 
only features which result from a reference graph. 

In addition, the sub-bunch jets of the reference bunch graph or graphs may also com- 
prise mixtures of these two above-mentioned features. 

These different comparison functions permit a use-oriented optimization of the method in 
such a way that the highest possible recognition rate and the highest possible speed 

□ are achieved. 

SI 

CTi According to a preferred embodiment of the above-described method, a step can be pro- 
vided according to which, after the recognition of each structure, the significance of the 
recognition is determined. For this purpose, an estimator can be used by way of exam- 

« pie, said estimator taking into account the optimum graph comparison function as well as 

« the non-optimum graph comparison function. 

■ rt 

□ An estimator in the case of which the distance of the values of the non-optimum graph 
comparison functions from the value of the optimum graph comparison function is deter- 
mined is particularly suitable for this purpose. 

By means of these measures, object and structure recognition can be achieved and in- 
formation on the quality of the structure recognition can be obtained as well. 

According to another advantageous further development of the above-mentioned methods, 
each structure can be associated with the reference images corresponding to the refer- 
ence graphs and/or the reference graphs from the reference bunch graphs for which the 
values of the graph comparison functions lie within a predetermined range. If the values 
do not lie within a predetermined range, this means that a structure cannot be identified 
sufficiently. Hence, this further development is suitable for cases of use where decisions 
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are to be taken on the basis of the recognition process, e.g. in the case of access con- 
trol. 

According to an advantageous embodiment, the colour information used in the above- 
described methods may comprise hue values and/or colour saturation values and/or 
intensity values determined from the reference image data and the Image data, respec- 
tively. 

Although the reference graphs and the reference bunch graphs, respectively, can be re- 
calculated prior to each application, which will be expedient in the cases of use where 
the reference data change frequently, especially due to updating, it will be expedient in 

Q most cases of use that the step of providing the reference graphs and the reference 
bunch graphs, respectively, comprises fetching the reference graphs and the reference 

Jj n s bunch graphs from a central data base and/or a decentralized data base, e.g. from chip 

l'O cards. 

a 

; s According to a preferred embodiment, the net-like structure of the reference graph can be 
:*f| used in the above-described methods in the form of a regular grid whose nodes and links 
1 1; define rectangular meshes. 

'sk* 

Q 

f " Alternatively, an irregular grid can be used as a net-like structure of the reference graph, 
the nodes and links of said irregular grid being adapted to the structure to be recog- 
nized. The nodes can have associated therewith characteristic points, so-called land- 
marks, of the structure to be recognized. 

It follows that, in the case of this further development, the jets are determined at the 
characteristic points of the structure. This means that, when the image data and the ref- 
erence data are compared, the characteristic points are primarily taken into account, 
whereby the significance with which a structure is recognized can be increased. 



According to a preferred embodiment, Gabor filter functions and/or Mallat filter functions 
can be used in the above-described methods as class of filter functions for convolution 
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with the reference image data and image data, respectively, and/or as class of filter func- 
tions for convolution with the colour-segmented reference image data and Image data, 
respectively. 

According to a preferred embodiment of the above-described methods, the projection of 
the net-like structure of the specific reference graph and/or the specific reference bunch 
graph may comprises centering the reference graph and/or the specific reference bunch 
graph in the image. 

In addition, it proved to be advantageous when the projection of the net-like structure of 
the specific reference graph and/or of the specific reference bunch graph comprises a 

i;3 displacement and/or rotation of the centered reference graph and of the centered refer- 

\j ence bunch graph, respectively. 

m 

i;Ct This will reduce the amount of time required for recognizing the structure. 

en 

" In particular, the projection of the net-like structure of the specific reference graph and/or 

|iB of the specific reference bunch graph may comprise scaling the centered reference 
; s graph and the centered reference bunch graph, respectively. This permits espedally an 
p increase in the significance and the speed of recognition, if the structure to be recog- 
nized differs in size in the image data and in the reference data 

The displacement and/or the rotation as well as the scaling of the centered reference 
graph and of the centered reference bunch graph, respectively, can in this case by car- 
ried out simultaneously, whereby the amount of time required for recognizing a structure 
can be reduced. 

In addition, the projection of the net-like structure may comprise local distortions of the 
centered reference graph. This embodiment will be particularly suitable for cases where 
the image data and the reference data have been recorded at different recording angles. 
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It will be expedient to cause such a local distortion by a local displacement of a respec- 
tive node of the centered reference graph. 

In accordance with an advantageous embodiment, the displacement and/or the scaling 
and/or the rotation can be determined on the basis of a comparison between the image 
graph and the corresponding reference graph and/or the corresponding reference bunch 
graph. This will lead to a substantial increase in the recognition speed. 

Further preferred embodiments result from the subclaims and from the description of pre- 
ferred embodiments of the method according to the present invention, which will be de- 
scribed hereinbelow making reference to the drawing, in which: 

Q 

\Zi Fig. 1 shows an image of a hand position with a reference graph for explaining one 
embodiment of the present invention; 

rU 

;;3 Fig. 2 shows the reference graph according to Fig. 1 ; 

□ Fig. 3 shows a schematic representation for determining a graph from a reference 
j*f image according to one embodiment of the present Invention; 

p! Fig. 4 shows a schematic embodiment for explaining a further embodiment of the 
present invention; 

Fig. 5 shows images of hand positions from which a reference data base has been 
ascertained; 

Fig. 6 shows examples of images of a hand position with different backgrounds which 
were compared with the images of Fig. 4 by means of one embodiment of the 
present invention. 

In the following, the various embodiments of the present invention will be described on the 
basis of a method for recognizing the position of a hand, referred to as hand position here- 
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inbelow. This description must, however, not be regarded as a restriction, but only as an 
example of a method for recognizing structures and objects, respectively, in images. 

Fig. 1 shows, by way of example, an image 20 of a hand at a certain position. 

For carrying out the method according to the present invention, the image 20 must exist in 
digitized form. The digitized form can either directly result from the recording technique in 
question, e.g. when a CCD camera is used, or it must be obtained by converting an analog 
picture, e.g. a conventional photo. 

A digitized image exists typically in the form of a pixel field having a predetermined size. 
Each pixel has associated therewith a horizontal and a vertical position x . Hence, pixel x 
•£j stands hereinbelowfor the pixel which has had associated therewith the position x . In ad- 

gi dition, each pixel has assigned thereto a grey value and/or a colour information value, such 
i*U as an HSl value. 

m 

1 Local image information, so-called features, which are, in turn, combined to form so-called 
Q jets, is now extracted from the digitized image 20 and/or from the image data corresponding 
2 to said digitized image within the framework of the method according to the present inven- 
ts tion. These extracted features are arranged in the form of a graph. 

The predetermined pixels are obtained in that a net-like structure 21 , which, for the sake of 
simplicity, will be referred to as grid in the following, is first projected into the image 20. 

As can be seen from Fig. 2, in which this grid 21 is shown in detail, the grid used in the first 
embodiment comprises 15 nodes 22a, 22b, 22c, ... and 19 links, i.e. connections between 
two respective nodes. In Fig. 2 the links between the nodes 22a and 22b, and 22b and 22c 
are designated by reference numerals 23ab and 23bc, respectively. 

Finally, the predetermined pixels are obtained by determining the pixels corresponding to 
the projection of the nodes. 
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In the first embodiment an object-adapted grid is used, i.e. the nodes are associated with 
characteristic points in the image. For recognizing the position of a hand, the nodes are 
therefore assigned to the two fingers and the back of the hand, as can be seen in Fig. 1 . 

The present invention is, however, not limited to such object-adapted grids, but it is also 
possible to use grids in the actual sense of the word, Le, regular grids. The decision which 
form of grid will be expedient, depends on the respective case of use. In comparison with 
regular grids, object-adapted grids will normally lead to a more significant recognition, but 
they are more difficult to handle. 



Fig. 3 shows a schematic representation on the basis of which it is explained how a repre- 
sentation in the form of a graph is obtained from image 20; as will be explained hereinbe- 

'SET? 

,3 low, this representation in the form of a graph can be compared with other graphs, in par- 
~2 ticular with reference graphs. 

m z 

;y 

q When specific pixels have been determined by the grid, as can be seen in Fig. 3(a), the 
1 ^ features in these pixels are extracted as described hereinbelow. 

In the first embodiment two different classes 28 and 29 of features are used for comparing 
!j5 the image with a reference image. 

The first class of features 28a, 28i is defined as the result of a convolution of the image 
at a predetermined pixel with a given filter function. 



According to the first embodiment, so-called complex Gabor filters are used as filter func- 
tions for this purpose. These filters can be represented by the following formula: 



k 2 

^(x) = -\ exp 
a 



I" 2a 2 J 



exp(ik r x)-exp | — 



,2\ 



(1) 



The Gabor filters represented by equation (1) have the form of a plane wave described by a 
wave vector kj and limited by a Gaussian function having a width a / k, wherein a - 2%. By 

selecting the wave vector kj the magnitude and the orientation of the filter function can be 
determined. 

In addition, the filter functions satisfy the condition: 

Iv, = 0. (2) 

X 

in Fig. 3 (c) the real parts of such two different filter functions are shown, cf. 24a and 24b. 
The value 0 is represented by a medium grey; positive values are brighter and negative 
M values are darker The filter function 24a shown in Fig. 3(c) has a low frequency and a small 

psj wave vector k } with an orientation of approximately 60 degrees relative to the horizontal. 

Q The filter function 24b has a higher frequency and a larger wave vector k.with an orienta- 
n tion of approx. 90 degrees relative to the horizontal. 

^ It follows that for a predetermined wave vector k^ i.e. by selecting the magnitude and the 
^ orientation of the filter function, the feature Jj(x) can be calculated at a predetermined pixel 
x by: 



= 2^)v,(x-5?). (3) 

X* 

I (x) stands for the intensity in this equation. 

In dependence upon the wave vector kj , various features J^x) can now be calculated for 

each pixel x , said various features being combined in a sub-jet for the pixel x In the first 
embodiment 
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In the images in Fig. 3(d), which are designated by 26a and 26b, respectively, the results of 
the convolutions of image 20 with the filter functions 24a and 24b are shown for all pixels x . 

When the wave vector k, is represented by: 
- (Kcos^ -zl n 

k i = U v siN,J' kv = 2 2 W 

wherein v is the magnitude index and n is the orientation index of the filter function, the 

features for the indices \i e {0,..., 7} and v e {0 4} are combined in a sub-jet 28 of jet 27 

in accordance with the first embodiment 

C J5 It follows that the sub-jet 28 for the pixel x comprises a total of n=40 features. 

rii 

q In Fig. 3 part of these features is schematically shown in the sublet 28. Features 28a, 28b 
m and 28c were obtained by means of filter functions having a constant magnitude and, as 
q can be seen from the hatching, different orientations. The same applies to features 28d, 
g 28e and 28f as well as 28g, 28h and 28i. In comparison with features 28a, 28b and 28c, 
53 features 28d, 28e and 28f as well as 28g, 28h and 281 result from a convolution with a 
! ? smaller filter function. 

In addition to the above-described Gabor filter functions, arbitrary other filter functions can 
be used as well. According to a further embodiment of the present invention, so-called 
Mallat filters are used as a further example of a class of filter functions. 

The convolution of the image with these Mallat filters can be expressed by the following 
formula: 



T„(x) = Kx> 



■ w (xj* 



or?'* 



(x)J 



(5) 
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wherein * stands for the convolution operation, h and v stand for horizontal and vertical, and 
St (si = So2', i e N) represents the width of a Gaussian function whose derivatives are used 
as filter functions. 

The second class of features 29a, .... 29i is defined as the result of a convolution, at a pre- 
determined pixel, of data with a given filter function, said data resulting from a conventional 
colour space segmentation of the image data. In the present case, the colour space seg- 
mentation was carried out with respect to the skin colour. It goes without saying that, de- 
pending on the respective case of use, the colour space segmentation can also be carried 
out with respect to other colours. In Fig. 3(b) the image data 22 are shown after the colour 
space segmentation. 

•its? 

;. gi In the present embodiment, the convolution of the colour space-segmented data was car- 
;■:! ried out analogously to the convolution of the image data, i.e. with the same Gabor filters of 
rtl the same magnitude and orientation. Hence, a detailed description of this convolution op- 
j=| eration can be dispensed with; in this connection, reference is made to the respective para- 
Cfi graphs dealing with the convolution of the image data with the Gabor filters. 

t SB | 
p.: 

JJJ It should, however, additionally be pointed out in this connection that in Fig. 3(e) the colour 
Ijjj space-segmented image data folded with the filter functions are shown. The images 25a 
and 25b in Fig. 3(e) represent the real parts of a convolution of the colour-segmented image 
data with the filters whose real parts are designated by 24a and 24b in Fig. 3(c). 

Analogously to the features resulting from the convolution of the image data with the Gabor 
filters, also the features obtained by folding the colour space-segmented image data with 
the Gabor filters are combined in a sub-jet 29. Since the magnitude index and the orienta- 
tion index of the filter functions have been chosen in this embodiment as described above, 
the sublet 29 also comprises n=40 features for the pixel x , the features 29a, 29i being 
shown representatively. 

It goes without saying that, instead of the Gabor filters, other filter functions, such as the 
Mallat filters which have already been mentioned, can be used in this case as well. 
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The sub-jets 28 and 29 define the jet 27 in the first embodiment The jet 27 finally com- 
prises 80 features in this embodiment 



When the extraction of the features is carried out at all points of the grid, the desired graph 
will finally be obtained, said graph reflecting the structure of the grid 21 and comprising the 
jets at the points corresponding to the grid points. 

According to a further embodiment colour information obtained from the image data, such 
as the hue, colour saturation and intensity (HSI), is used, instead of the convolutions of the 
filter functions with the colour-segmented data, as second class for the pixel. According to 
this embodiment, a single jet is therefore composed of 40 features resulting from the 
above-described convolution of the image data with the Gabor filter functions and of the HSI 

**3 triple; hence, it comprises 43 features. 

'SI 

tji 

m It will be expedient to used a weighted Euclidian distance in the HSI colour space as a 
^ comparison function between two HSI triples. 

5 

P In addition, a jet is composed of three classes in the case of a further embodiment. The first 
^ class comprises the features resulting from the convolution of the image data with the Ga- 
s,p bor filters (Gabor features); the second class comprises the features resulting from the 
convolution of the colour space-segmented image data with the Gabor filters (colour-gabor 
features); and the third class comprises the HSI triple obtained from the image data (colour 
information). 



According to a further embodiment, a texture description gained by statistical methods can 
be used alternatively to or in addition to the above-described classes. Such texture descrip- 
tions are e.g. the medium grey value, the variance of the grey value distribution, the covari- 
ance matrix of the grey value distribution, the entropy, the orientation of the grey value 
structure, medium scales of the structure in various directions, the variation range of the 
local orientation, the variation range of the spatial scales, the sequence and the arrange- 
ment of various elements of the structure. 

In addition, and again alternatively to or in addition to the above-described classes, motion 
vectors (displacement vectors) are defined as a class according to a further embodiment. 
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Such motion vectors can be calculated from two successive images by means of differential 
geometric methods (differential methods), correlation methods and filter methods. Further- 
more, it is also possible to calculate motion vectors from two Gabor jets. 

In addition to the embodiments described in detail, a large number of combination possibili- 
ties of the above-described classes exist in dependence upon the field of use, I.e. the 
structure, the quality and the complexity of the content of the image data to be compared. It 
is especially also possible to use the features of only one of the above-described five 
classes for the comparison. In particular, it is also possible that features resulting from con- 
volutions of the image data with a first filter class define the first class, and that features 
resulting from convolutions of the image data with a second filter class define the second 
class. 

a 

sj In the following it will be described how an image is compared with a reference image. The 
fi| first word "comparison" in compound words refers to the respective image recorded in 
which the structure or the object is to be recognized. 

S 

^ To begin with, a reference image in the form of a graph, i.e. in the form of a reference 

If graph, must be provided. This can be done e.g. by fetching the reference graph from a 
; I central data base or a decentralized data base, e.g. from chip cards. 

When the data base is being created, this reference graph G is determined according to 
one of the above-described methods. In the present case, a grid of the type shown e.g. in 
Fig. 2 is therefore projected into the reference image. At the nodes of the grid, the jets are 
determined, as has already been explained hereinbefore as well. The jets of the reference 
graphs should comprise at least the classes of features which are to be determined for the 
comparison graph as weii. 

In the next step, the grid of the reference image (reference grid) is projected into the com- 
parison image and the jets corresponding to this grid are calculated for the comparison im- 
age. In so doing, filter functions, i.e. the same class of filter functions, the same magnitudes 
and/or the same orientations, with which the reference graph has been created should be 
used at least partially. The thus calculated jets together with the structure of the projected 
grid will finally define the comparison graph G'. 
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In dependence upon the differences between the reference Images and the comparison 
images to be expected, various mappings can be used for projecting the reference grid into 
the comparison image. 

The simplest projection, viz. centering the reference grid in the comparison image, will e.g. 
be suitable for cases where the reference image and the comparison image have the same 
size and the same position with respect to the image centre and have been recorded at the 
same angle. 

In addition to this simple projection, the following projections can be carried out selectively, 
!]g Displacement of the reference grid in Its entirety: 

J this mapping will be suitable for cases where the reference image and the comparison im- 
py age occupy different positions with respect to the image centre. 

3 

ffl Scaling of the reference grid: 

%i this mapping can be provided in cases where the reference image and the comparison im- 
; S age are different in size. 

:J=t 

Rotation of the reference grid: 

a rotation can be used in cases where the reference image and the comparison image are 
rotated relative to one another. 

Local distortion of the reference grid: 

in the case of this mapping individual points of the grid are displaced relative to their posi- 
tion in the reference grid. This mapping is particularly suitable for cases where it is to be 
expected that the comparison image shows local distortions in comparison with the refer- 
ence image. 

It goes without saying that arbitrary combinations of the above-described projections are 
possible as well. 
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The evaluation of the similarity of the two graphs is carried out by means of a graph com- 
parison function of the following general form: 

S = S Jet +X£ MeWl . (6) 



Sjrt stands here for the jet comparison function, i.e. a suitable function which evaluates the 
similarity of the jets at corresponding points of the two graphs, and S Me «k stands for a suit- 
able function which compares the similarity of the metrics of the two grids. Su«*k depends 
strongly on the projection used. 

P, A, (A, & 0) designates the weighting of the two comparison functions relative to one another. 
™j X can also be made equal to zero; this means that the similarity of the metrics of the graphs 

tji is not taken into account This value will be particularly expedient in cases where only cen- 
j tering or displacement Is chosen as projection or, In other words, where the topologies of 
Q the reference graph and the comparison graph are identical. 

S3 For calculating the jet comparison function Sj* it will first be necessary to calculate the 
|,* comparison functions of the sub-jets for the respective classes k. In the case of the first 
*0 embodiment, whose features result from a convolution of the image data and of the colour 
M= space-segmented data with Gabor filters, k would be equal to 2. 

For evaluating the similarity of two corresponding sub-jets j k and j k ' of the jets j and j' of the 
reference graph G and of the comparison graph G', respectively, a sub-jet comparison 
function is used, Le. a function which depends on the amplitudes aj and a/ of the two sub- 
jets and which has the following form: 

The present invention is, however, not limited to this comparison function; it is also possible 
to use a phase-sensitive comparison function having e.g. the following form: 
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2>ja;cos(<|>, -ty -8-kj) 



wherein kj is the wave vector of the respective Gabor filters and d is an estimated 
displacement vector which compensates fast phase shifts, d is determined in that 
is maximized in its Taylor development within a small square, which is centered in 
d = 0. The term dkj with the estimated displacement vector d finally compensates 

fast phase shifts on the basis of small variations of the positions x and ? of the two 
Q jets compared with one another. 

'Si 

iXf ■ 

ru a j(*) and *j09 result in ^ s from J j(*) = 8 jP) wf^jt^O) • 



•tSS* 



Q A single jet comparison function for the respective individual jets can be formed 
l«& making use of the sub-jet comparison functions for the sub-jets corresponding to one 
another, i.e. for the features of a class: 

saJ , )=n({s lt (j k ,j ll ')}). (9) 
Depending on the respective case of use. different functions n can be used, for example 



S k (j k ,jV)}) = Z fi >A(j k .JV),or (10) 

k 

"({S k (j k ,j k , )}) = n(S k (j k .iH , )r. or (11) 

k 

«({S k (j k , V)}) = max{(D k S k (j k , V)} (12) 

^{S k (j fc J,« , )}) = min{<D l( S lt (i 1|1 j k ')} (13) 
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S k (j k . j k ' ) are here the sub-jet comparison functions which can be calculated accord- 
ing to equations (7) or (8); ® k are weighting coefficients which describe the contribu- 
tion of a single sublet to the single jet comparison function S(j, j' ) . 

The jet comparison function Sj* for all the corresponding jets of a reference graph G 
and of a comparison graph G' is finally formed from the single jet comparison func- 
tions for the individual jets. For this purpose, different functions can be used, again in 
dependence upon the respective case of use. The comparison function for n jets can, 
for example, be formed according to: 

S J «(G,C) = 2> n S n <J n ,j;),or (14) 



4 S M {G,G) = n< S n(«U„')r. (15) 

!;fl n 



ru 

CO 



a 



S n( J n . «V ) ar e here the single jet comparison functions which can be calculated ac- 
cording to equations (10) to (1 3); a>„ are weighting coefficients which describe the 
contribution of a single jet to the jet comparison function . 

The magnitudes of the differential vectors of two corresponding links, i.e. of Hie con- 
nections between two corresponding nodes, can, for example, be added so as to 
obtain the function for comparing the metrics of the two graphs; hence, the following 
metric comparison function is obtained for graphs with E links, which are referred to 
as Ax, and Ax,', respectively: 



SwCQG 1 ) = ^(A^-Ax') 2 



(16) 



Which of the comparison functions is used in detail and, in particular, what factor is chosen 
for X depends essentially on the structure of the reference graphs and of the comparison 



23 



graphs, j.e., in the final analysis, on the structure of the comparison images and of the ref- 
erence images. 

The selection of the suitable comparison function for a given structure of the comparison 
images and of the reference images can be determined by means of comparison tests with 
the comparison images and the reference images and does therefore not exceed the scope 
of the normal ability of a person skilled in the art. 

The comparison graph can now be optimally adapted to the reference graph with the aid of 
the graph comparison function (6). For this purpose, the projection of the reference grid into 
the comparison image is varied until the graph comparison function assumes an optimum 
value (in the case of the above-described comparison functions this optimum value is a 
minimum). 

• SI 

m 

i'U It is, of course, also possible to compare several reference images with the comparison im- 
| age. 

q These reference images may exist in the form of a data base. In this case it will, however, 
sH be necessary to calculate the reference graphs again for each comparison. 

j =f It will therefore be more expedient to provide the data base in the form of a reference graph 
data base from the very beginning, the reference grids with their respective jets being 
stored in said reference graph data base and being simply fetched therefrom. Such a refer- 
ence graph data base is, however, not as flexible as a reference image data base, since it 
has to be recompiled for each change in the calculation of the jets. 

For comparing the comparison image with an arbitrary number of reference images, the 
optimum adaptation to each reference image and the graph comparison function for mis 
optimum adaptation are determined for the comparison image by means of the above- 
described method. 

On the basis of a comparison between the respective graph comparison functions corre- 
sponding to the optimum adaptations of the reference images to the comparison image, the 
reference image which is most similar to the comparison image can be ascertained. 
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In addition, by evaluating the graph comparison functions for all reference images it will be 
possible to determine a measure of the significance of the respective similarity. According 
to the necessary degree of similarity, different definitions for a significant recognition can be 
used in this connection. 



For example, the average S and the variance c s can be formed from all the graph 

comparison functions for the non-optimum reference graphs. A significant similarity 
could be assumed if 



S ^s (17) 



or 



f *! 

m 
a 

m 



S °" S * £s (18) 



9 

n 



is fulfilled, wherein s is fixed parameter chosen and wherein S 2 is the second smallest 
value of all graph comparison functions. 



Fig. 4 shows a further embodiment of the present invention. 

This embodiment differs from the hitherto described embodiments with respect to the 
structure of the reference graphs which are compared with the comparison graphs. 

Whereas in the case of the hitherto described embodiments a reference graph has been 
created on the basis of a single reference image, a reference bunch graph results from a 
plurality of reference images in the case of this embodiment 



For this purpose, M model graphs are created on the basis of M images, said model graphs 
being combined in a so-called reference bunch graph. 

All M model graphs have the same qualitative structure, i.e. they have N nodes which are 
linked by a predetermined grid In this connection, it is, however, admissible that the lengths 
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of two corresponding links are different Hence, it is only demanded that the model graphs 
are topologically identical. The structure of the grid may have, a regular form, as shown in 
the present connection, or an irregular form. 

Hence, it is in particular possible to use the grids that have been used in the above- 
discussed embodiments, i.e. a regular nxm grid or an irregular object-adapted grid. 

In addition to the lengths of two corresponding links that differ, also the corresponding jets 
of the M model graphs are different 

The M model graphs are finally combined so as to form a bunch graph, as will be explained 

... hereinbeiow. 

u 

ixi To begin with, the mean distance vectors A, between the nodes i and j in each bunch 

IV' 

;;o graph are ascertained by: 

O. 

m 

a A * = nZ^. <1«> 

□ wherein A" is the distance vector between the nodes i and j in the model graph m. 

These distance vectors now determine the structure of the bunch graph which is af- 
terwards projected into the comparison image for the purpose of comparison. 

The nodes of the bunch graphs have additionally assigned thereto bunch jets of the 
M model graphs. These bunch jets comprise all sub-jets which have been ascer- 
tained during the creation of the respective reference graphs. Hence, a jet of the 
bunch graph comprises the M corresponding jets of the M model graphs. In other 
words, a jet of the reference bunch graph is composed of the respective sub-jets cor- 
responding to the sub-jets of the model graphs. 
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The resultant structure is schematically shown in Fig. 4. Gt,..., Gm designate here the 
M model graphs which are combined so as to form the bunch graph B M 40. 

For carrying out the comparison the bunch jets are associated with sub-bunch jets. 
Criteria which can be used as association criteria are e.g. the feature classes, the ref- 
erence images or mixed forms of these two. In the case of an association according to 
classes of features, the sub-bunch jets only comprise the features of a single class of 
features. 

For comparing a bunch graph B M 40 obtained in this way with a comparison graph G' 45 
resulting from an image 42, a bunch graph comparison function which is analogous to 
equation (6) is used according to this embodiment. 

. 

£fjj The jet comparison function is in this case determined in the way explained herein- 
ru below. 

y 1 To begin with a comparison function between a sub-jet j' of the image and the corre- 
Q sponding sub-bunch jets b M is determined. For this purpose, the following equations, 
t J which are analogous to equations (10) to (1 3), can be used by way of example: 

(20) 

(21) 

(22) 
(23) 

f is here a sublet of the image and j" is the corresponding sub-bunch Jet; the 
S„(j", j') are determined e.g. according to equation (7) or (8). 



S(b",j I ) = 5: o >nS„(j!f.j , ),or 

n 

S(b M J') = n(S„(CJ , )r,or 

n 

S(b M ,j')=max{© n S B (in.j , )}.or 
S^f^min^Ati"]')}- 
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A functional relation Q exists between the comparisons of the sublets and the sub- 
bunch jets on the one hand and the comparison function for the jet J with the bunch 
graph B M on the other 

S(B M . J ) = n({S M (br . j,' )}) =: n<M) • (24) 

The function SI can in this case be determined according to the following recursion: 

(25) 

(26) 

(27) 
(28) 



(29) 

(30) 

(31) 
(32) 

Depending on the respective case of use, the recursion can be carried out until the 
following condition is fulfilled for M (m> : 

|M< m) | = 1. 



ilri 

tu 



fi (0) (M) = 2;(D i n[ 1) (M l (15 ),or 

1 

n<°>(M) -IJ(nfW>r. or 

fi w (M) = max^Qf'W)} . or 
n (0) (M) = man|o,Q, w (M, (1) )| , wherein |J M i 1) = M > and 



^(m-D^I-t)) = 2]®jflj n,) (Mj m> ) , or 

j 

f2, (m " 1) (M( m " t> ) = max{© j m) (M| m> )} , or 
n (^D (M (^D) = r! ,in{ 0j nW(M J w )} , wherein [)M 



(m) =M (m-t> 
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Making use of the comparison functions for the jet J with the bunch graph B M for the indi- 
vidual jets, the jet comparison function Sj* for all the corresponding jets of a reference 
bunch graph B M and of a comparison graph G' fe finally formed. For this purpose, various 
functions can be used, again in dependence upon the respective case of use. The com- 
parison function for n nodes can, for example, be formed according to: 

s«(Br l G)»2«A( B ?.JL , ).« 03) 

n 

S Jel (B M ,G) = n(S n (B^J n ')r. (34) 

n 

This comparison function can finally be inserted in equation (6) for determining the 
9 graph comparison function. 

!» | Expressed in words, the jets having the maximum similarity are selected from the 

ffi bunch graph B M t when said bunch graph B M is compared with a comparison graph G. 

Q 

m 

!L In Fig. 4 this fact is expressed by arrows which extend from the respective node of 
the comparison graph G and which end at different model graphs. 

P In addition to the above-mentioned bunch graph comparison function also other suit- 
able comparison functions can be used. 

When a bunch graph is used, this bunch graph will, just as in the case of a compari- 
son with a single reference graph, first be projected into the image of the comparison 
graph and, subsequently, the optimum projection will be determined by evaluating the 
bunch graph comparison function. 

Since a bunch graph can be treated like a single graph, it is also possible to combine a 
plurality of reference bunch graphs, one reference bunch graph and one or a plurality of 
reference graphs in a data base with which the comparison graphs are to be compared, 

Making reference to Fig. 4 and Fig. 5, a concrete example of hand position recognition in 
front of a complex background according to the above-described method will be de- 
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scribed in the following, said method making use of features which result from a convolu- 
tion of the image data with Gabor filters (Gabor features), from a convolution of colour- 
segmented image data with Gabor filters (colour-gabor features) and from the HSI infor- 
mation (HSI features). This example is compared with a recognition method in the case 
of which only a single class is taken into account i.e. the respective weighting coeffi- 
cients o K are equal to zero. 

In Fig. 5 the representative reference images are shown, which have been used for cre- 
ating reference graphs according to the above-described method. Each reference graph 
comprises 15 nodes and 20 links or connections. The nodes were manually distributed at 
anatomically significant points. 

a 

In Fig. 6 various examples of a hand position in front of different backgrounds are shown. 
Cn In total, 29 backgrounds have been used, among which five contained a high degree of 

skin colour, eleven a medium degree of skin colour and eight a small degree of skin col- 
Q our. 

u Making use of these backgrounds, 1Q00 different comparison graphs have been created. 

h net 

;S From the reference data, 1 8 reference graphs which contained Gabor features or colour- 
i I gabor features or HSI information were determined for each of the hand positions shown 

in Fig. 5. Gabor filters with three different magnitudes and eight different directions were 

used for the convolution. 

From the respective 18 reference graphs three bunch graphs, each comprising 6 
models, were formed. The features of the sub-jets were used as a criterion for the as- 
sociation of the sub-jets with the reference graphs. Accordingly, the first reference 
bunch graph contained ail Gabor features, the second reference bunch graph con- 
tained ail colour-gabor features and the third reference bunch graph contained all HSI 
features. 

Upon projecting the reference bunch graphs, a rotation of up to 15°, a scaling of up to 
20% and a local distortion by at most one pixel were admitted. 
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The graph comparison function was calculated by means of the equations (6; A, = 0 ), 
(33; with identical ©J, (24), (25; with o 0abor = 0,25, ©coiourGabor = 0.25,® ^ = 0.5), (33; 
with identical ©J, (8) for the Gabor and colour-gabor features and a weighted Eucli- 
dian distance for the HSI colour information. 

The results of the comparison of the 1000 comparison images are summarized in the 
following table: 



Ms 



weighting simple background complex background 
only Gabor features 82,6% 70,4% 
only colour-gabor tea- 39,7% 34,6% 
tures 

only HSI features 88,2% 76,3% 

optimum weighting 92,9% 85,8% 



